Intrusion detection via semantic fuzzing and message provenance

ABSTRACT

Intrusion detection systems and methods monitor legal control messages in an operational control system to detect subtly malicious sequences of control messages with undesirable emergent effects on devices in the operational control system. A message provenance component may investigate system-level correlations between messages rather than detecting if individual messages are anomalous. A semantic fuzzing component may search, based on the operational effect of candidate message sequences, the space of legal messages for sequences that cause actual harm. Behavior oracles may be used to test message sequences to identify sequences that induce drift towards a failure state. The intrusion detection system is able to prevent harm and disruption arising from control messages that individually appear legitimate and benign but that, in combination with other messages, can cause undesirable outcomes.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/308,563 filed on Mar. 15, 2016, and entitled“Semantic Fuzzing and Message Provenance,” and U.S. Provisional PatentApplication Ser. No. 62/318,420 filed on Apr. 5, 2016, and entitled“Semantic Fuzzing and Message Provenance.” The disclosures of theseprovisional patent applications and references cited therein are herebyincorporated by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

Not applicable.

FIELD OF THE INVENTION

The invention relates generally to detection of intrusion by subtlymalicious control messages and data packets in operational controlsystems, and more specifically, to the identification of threatsinvolving legitimate-looking control and data messages that give rise toharmful, unpredictable, or otherwise undesirable emergent behaviors in,for example, the devices of industrial control systems.

BACKGROUND

Typical cyberattackers, as part of a cyber campaign, cyberwarfare, orcyberterrorism, may seek to introduce a malicious computer program(“malware”) into a computing system for purposes of spying or causingdamage or disruption to a target. Conventional intrusion detectionsoftware like firewalls monitor files for anomalous data that has beenidentified as being a threat. Skilled cyberattackers, on the other hand,may use messages that combine expected and allowed operations in noveland unanticipated ways to evade observation. Industrial control systems(ICSs) and operational technology (OT) systems are particularlysusceptible. Carefully crafted sequences of legitimate-looking messages,which individually may be benign, can give rise to emergent behavior,and such unpredictable behavior may severely damage or destroy, forexample, the ICSs that operate and monitor the energy grid. Existingmethods of intrusion detection using signatures and content anomalydetection lack the ability to perceive the latent harm in such messages.

SUMMARY

Exemplary embodiments of intrusion detection and protection systems andmethods (collectively, “systems”) are able to monitor legal controlmessages in an operational control system (such as an industrial controlsystem (ICS)) to identify subtly malicious sequences of control messageswith undesirable emergent effects on devices in the control system. Amessage provenance component may investigate system-level correlationsbetween messages rather than detecting if individual messages areanomalous. A semantic fuzzing component may search, based on theoperational effect of candidate message sequences, the space of legalmessages for sequences that cause actual harm. Behavior oracles may beused to test message sequences to identify sequences that induce drifttowards a failure state. The intrusion detection system is able toprevent harm and disruption arising from control messages thatindividually appear legitimate and benign but that, in combination withother messages, can cause undesirable outcomes. Further advantages andfeatures of the invention will be apparent from the remainder of thisdocument, which discusses various exemplary implementations, inconjunction with the associated drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates several exemplary options for integration ofexemplary detection systems in a control system.

FIG. 2 depicts an exemplary architecture for a system for detection ofand protection from subtle malicious messages. The capture of globalcontext information provides an informed basis for encoding,representing, and querying the actual legitimate state of a specificsystem being protected (such as an energy system). The context systemsupports the detection techniques of message provenance and semanticfuzzing and informs the operation of a collection of high-fidelityoracles that emulate various aspects of ICS equipment behavior andphysical properties.

FIG. 3 provides an overview of an exemplary message provenance system.Messages may be extracted for each connection pair and each protocol.Correlation between different connections can be extracted, andprobabilistic models derived for the linked connections. The models areused by the behavior oracle(s). In a testing phase, new messagesequences may be fed to the oracle hierarchy to determine an anomalyscore.

FIG. 4A provides an example of a regular sequence of commands pattern.The y-axis presents the command type and the x-axis the index of eachcommand.

FIG. 4B provides an example of a relatively less-regular commandspattern. As with FIG. 4A, the y-axis presents the command type and thex-axis the index of each command.

FIG. 5 provides an overview of an exemplary semantic fuzzing component.A message generator generates possible message sequences based on thecurrent context, state, and message history. These messages are testedagainst the behavioral oracles to determine sequences that induce drifttoward a failure state.

FIG. 6 is a depiction of an exemplary semantic fuzzing framework feedingmessage sequences to behavior oracles.

The foregoing and other aspects and advantages of the invention willappear from the following description. In the description, reference ismade to the accompanying drawings that form a part hereof, and in whichthere is shown by way of illustration preferred embodiments of theinvention. Such embodiments do not necessarily represent the full scopeof the invention, however, and reference is made therefore to the claimsand herein for interpreting the scope of the invention.

DETAILED DESCRIPTION

Determining whether “legal” messages are benign or malicious poses anoperational challenge in the effort to safeguard against cyberattacks.Legal messages are messages that are valid with respect to somespecification. They may contain operationally valid commands and/or datathat affect the operation of one or more devices in the system.Industrial control systems (ICSs), information technology (IT) systems,and operational technology (OT) systems, for example, those in theelectrical power domain, can be targeted and exploited bylegitimate-looking control and data messages. This is a class of attacksthat differs from code-injection exploits against programmable logiccontrollers (PLCs) or supervisory workstations. The messages used arewell-formed and contain valid parameters and field values; any singlemessage may be completely legal in both syntax and semantics. However,the emergent properties of a set or sequence of such legitimate,well-formed messages can damage or disrupt a system. If any singlemessage looks legitimate, traditional intrusion detection systems andfirewalls will fail to match it against their signatures and rulesets.What is needed is an effective way to recognize and neutralize subtlymalicious control and data messages.

Exemplary intrusion detection systems assess the effects of the message(or message sequences) with respect to normal operations of the systemas a whole to identify deviations and undesirable behaviors. Gatheringand representing system context provides “ground truth” about how thesystem is or should be operating. The intrusion detection system may, inexemplary implementations, involve: (1) gathering, representing, andanalyzing system context, (2) mapping message provenance between groupsof nodes, and (3) using semantic fuzzing to assess the actual impactcurrent and future messages will have on the system. A challengecontemplated by gathering context is that, in the intrusion detectiondomain, “context” at any reasonable scale is often unstructuredinformation. In exemplary implementations, formal methods may be used tomodel context as a set of constraints on various system properties. Achallenge contemplated by mapping message provenance is that standardanomaly approaches fail because messages are not obviously anomalous. Inexemplary implementations, the relationships between messages andmessage field values may be modeled, and system-level correlationsbetween messages may be investigated, rather than detecting ifindividual messages are anomalous. And a challenge contemplated bysemantic fuzzing is that typical fuzzing approaches focus on perturbingthe syntax of messages or data; naive versions do this blindly, and moreadvanced ones use the program logic. In exemplary implementations,semantic fuzzing searches, based on the operational effect of thecandidate message sequence, the space of legal messages for sequencesthat cause actual harm. As an overview, exemplary implementations of thesemantic fuzzing approach treats fuzzing as a search problem through aninput space. Semantic fuzzing may have a different evaluation functionthan prior fuzzing techniques. In certain configurations, the approachmay involve building or modifying an existing ICS protocol fuzzer,formally verifying that the fuzzer conforms to the grammar orstandard/specification, subsetting the fuzzer to delete generation rulesfor “illegal” transformations, and re-verifying. This can be used tocontinuously produce candidate message sequences with valid fieldvalues.

Referring to FIG. 1, a control system 100 may include controlsubstations 110, 115 in communication with intelligent electronicdevices (IEDs) 120, 125 via a substation local area network (LAN) 130 orother suitable communications channel or channels. The control systemmay be an ICS that is part of, for example, the energy infrastructure.The control substations 110, 115 may be used to distribute controlfunctions in the control system 100, and the IEDs may be any machine ordevice that is controllable via one or more of the control substations110, 115. In exemplary embodiments, an intrusion detection system couldbe deployed in multiple ways with respect to the control system 100. Forexample, the system could be implemented as an independent drop-indevice 140 that is networked so that it is able to communicate withother components of the control system 100 via LAN 130. The independentdrop-in device 140 could have one or more processors, and memory withinstructions that are executed by the processors, to implement theintrusion detection scheme (e.g., context gathering, mapping messageprovenance, semantic fuzzing, reporting, etc.) that is suitable for theparticular control system 100.

In alternative embodiments, the intrusion detection system could beintegrated with a deployment process of an IED 150. That is, hardware,software, and/or firmware may be provided or added to one or more of thedevices 120, 125 that are being controlled by one or more controlsubstations 110, 115 in the control system 100. In such embodiments, theintrusion detection system could use the same communications channelsavailable to the IEDs 120, 125 (or could be provided with additionalcommunications channels if desired). In other exemplary embodiments, theintrusion detection system could be collocated 160 with a controllersubstation. Similar to the integration with IED option 150, hardware,software, and/or firmware may be provided or added to one or more of thecontrol substations 110, 115 in the control system 100, and the samecommunications channels available to the substations 110, 115 could beused.

Additional features, details, and advantages will now be discussed inthe context of certain exemplary embodiments capable of detecting andresponding to crafted and subtle cyberattacks on, for example, energyICSs or other control systems. Such exemplary embodiments focus on theparticularly difficult challenge of identifying the latent harm insequences of seemingly benign or normal communications. They may use (i)system-wide context encoding mechanisms, (ii) novel message provenancemodeling techniques, and (iii) predictive semantic fuzzing processes,each of which will be discussed in more detail.

ICSs are inherently difficult to protect because they are targets topowerful adversaries who are capable of covert and subtle attacks. Asignificant threat arises from the emergent properties and latentbehavior induced via carefully crafted sequences of well-formed, legalmessages; both Stuxnet and the Ukraine incident (discussed below)included such control messages on the OT side of the attack. If therehad there been sensing elements in these environments that were capableof reasoning about the effects of such “benign” messages and theirinconsistencies with the broader operational context, such elementswould have contributed system resilience and defense-in-depth. A keychallenge arises from the difficulty of capturing and modeling arbitrarycontextual information about a complex cyber-physical system.

Exemplary embodiments may be based on two key tenets. First, whenindividual commands are all legal and well-formed, it can still bedetermined whether a command sequence is malicious by examining theglobal, system-level state. A novel message provenance technique may beused to discover unstated system-level dependencies between ICS devices.Second, starting from a given state, the system may look into the futureto identify unfolding command sequences that are likely to produceunexpected emergent behavior. This novel semantic fuzzing techniquegenerates possible sequences of future commands and tests them againstmodels of ICS behavior to determine if they result in an undesirablestate. The system, in effect, uses the beginning of an attack to predicthow it will unfold, using the attacker itself to help make suchpredictions.

Cyberattackers in this domain will have invested the time and effort tobypass traditional security measures, which are ill-equipped to contendwith techniques for disguising these attacks, as they operate attoo-fine a granularity (e.g., on a per-message basis), and are unable toproperly collect, assess, and interpret the larger context of messages.ICS equipment operates with a great deal of context, including systemsettings (policy) and configuration (setpoints), current environmentalparameters like load and time of day, maps of typical peer machines andstandard information flows, and the physical properties,characteristics, and limitations of devices.

In exemplary embodiments, context information is modeled in astandardized, coherent fashion that enables the identification ofadvanced attack encodings. A foundation of the effort to counter subtleattacks is to gather this information and create effective models fordistinguishing between legitimate inputs and attacks. This may beaddressed by creating a context collection system for information suchas configuration settings and physical properties to develop an initialrepresentation of legitimate system behavior. Based on this data,message provenance may then be used to discover implicit dependenciesand correlations from the patterns of communication among devices in avariety of settings (such as energy delivery devices that are part ofthe energy infrastructure).

The intrusion detection system preferably does not impede criticalfunctions (such as energy delivery). To be non-disruptive, obtainingaccess to meaningful data and communications information should bebalanced with the need to refrain from disturbing the operation of ICSdevices. The flexibility to be deployed as a monitor of communicationson an IT/OT subnet broadcast medium (in the case of protecting legacydevices) or integrated within new equipment to leverage on-board,independent, and compartmentalized computing capacity, facilitatesstriking an acceptable balance. A reporting capability in certainimplementations may fuse raw detection information into actionablethreat intelligence in a standard format, such as Structured ThreatInformation Expression (STIX)/Trusted Automated eXchange of IndicatorInformation (TAXII). This information will enable better targeting ofexisting remediation and recovery procedures toward the observed threat.The intrusion detection system enables asset owners to monitor risk, andvendors to develop measures to reduce risk of carefully crafted attacksthat could drive systems to their operational edge (a state beyond whichdisruptive or damaging behaviors too likely). A focus on the interfacebetween the cyber and physical layers can help prevent physical damageto equipment with potential cascading effects.

Exemplary embodiments of the intrusion detection system detectadversarial manipulation of components of control systems by targetingstealthy attacks based on clever arrangements of legal operations thatultimately lead to physical damage or other operational disruptions.Specifically targeted are attacks that combine expected, allowedoperations, in novel or unanticipated ways to produce targeted, novelphysical behavior that disrupts or destroys equipment. One applicationis energy delivery systems. Energy generation and delivery systems arevulnerable to, and have been targeted, by well-crafted cyberattacksaimed at both the commodity and ICS-specific parts of thisinfrastructure. Existing solutions use a traditional approach to networkintrusion detection systems (IDS), which relies on pre-definedsignatures to identify potential ICS attacks. However, such an approachis limited when faced with the zero-day attacks that have becomeprevalent in ICS. Such attacks can evade existing detection techniquesby using legal control messages and data packets (referred tocollectively as “control messages” or “control data”) in unexpectedways. Such message combinations (which take into account temporalaspects, rates, sequences, and data combinations) can give rise toemergent behavior, and this unpredictable behavior can severely damageor destroy ICS energy equipment.

As an example, in 2009, Stuxnet emerged as a significant public exampleof this type of attack on ICS equipment. Its impact was significant:1,000 out of 9,000 centrifuges at the Natanz Fuel Enrichment Plant weredestroyed. Stuxnet used two different strategies to damage thecentrifuge rotors: the first one aimed to over-pressurize thecentrifuge, and the second to over-speed centrifuge rotors to lead themto resonance. Stuxnet's newest version infiltrated as a legitimatedriver for Windows hosts, and is an example of legitimate messages andcommands, with “legal” parameter values, that were used to slowly shiftthe system into a state that damaged the centrifuges. Critically, thisattack was also designed to evade (or make irrelevant) normal safetyprecautions typically embedded in an ICS. For example, a sudden stop ofa centrifuge could result in catastrophic damage, but if such a commandwere to be issued, the frequency converters would likely prohibit such aradical maneuver. A more subtle approach may be to instruct a centrifugeto slow down, resulting in the frequency converter to smoothlydecelerate (like with an isolation/rundown event) and then tosubsequently resume normal speed. Operating at or above certain speedsmay cause the rotor to vibrate (if only briefly). When a rotor passesthrough such speeds, the harmonics may damage the rotor. Consequently,such changes in speed have a chance of causing damage, and withrepetition over time, become more likely to damage the equipment.Notably, such an attack does not aim to produce catastrophic physicaleffects (which would likely be intercepted by existing safety bypassesand controls) directly. Instead, it seeks to induce catastrophic damageas a side effect of “normal” commands.

Another example of such an attack was the 2015 Ukrainian incident, wherepower companies experienced outages that affected 225,000 customers.Multiple breakers were tripped by the attackers. Though all the commandswere legal, their sequencing led to severe physical consequences. Ineach of these cases, existing security measures failed because they wereonly afforded a local view of the system, limited in both scope andtime. Interactions between different components and devices wereunmonitored, and the effect of a specific command was not investigatedbeyond its immediate time horizon, in conjunction with the othercommands in its sequence.

A foundational component of exemplary embodiments of the intrusiondetection system is a context collection system, which involvescollecting, representing, and storing global “context” for devices(equipment, machines, etc.) in the system (such as energy grid ICSdevices). In this domain, context includes configurations,specification-based protocol models, architecture information, topology,a physical model (if available), etc. The intrusion detection system mayuse a model for representing this information in a standard, coherentfashion. The context collection system captures the system-wide contextto support complementary message provenance and semantic fuzzingcomponents/layers, both of which help identify abnormal communications:the former with respect to typical communications, and the latter withrespect to induced physical behavior.

The context collection system facilitates the collection, normalization,and modeling of ICS context, allowing information to be extracted forsystem- and protocol-wide analysis. It helps establish a reliable,independent model of the system's legal operation as defined byoperational settings and physical limitations. The message provenancesystem identifies legitimate/normal flows between devices and componentsto determine normal message sequences and their correlations at thesystem level. By moving beyond the consideration of individual messagesor flows, the message provenance layer is able to capture correlationrelationships between multiple pairwise connections across the network.The system moves beyond modeling individual ICS interactions to modelingdevice interactions at the system level. The semantic fuzzing systemidentifies potentially harmful sequences as they start developing byusing state and context information to generate predictive commandsequences and testing them against behavior oracles. The informationproduced by context modeling and message provenance can be used togenerate and test predictive message sequences. These sequences will begenerated by a semantic fuzzing component that uses a concrete seed tosynthesize a sequence of valid messages, and then evaluates them againstbehavior oracles, which determine the system effect of the messagesequence for the given real-time context. Advantageously, exemplaryembodiments of the intrusion detection system can be integrated withexisting industry standards and components for all system layers,including information collection, analysis, and reporting, providingactionable threat intelligence in a standard format (such asSTIX/TAXII).

A detailed exposition of technical components of the intrusion detectionsystem begins with the illustration of the overall exemplaryarchitecture shown in FIG. 2, which depicts a multi-layered approach fordetecting catastrophic global state shifts induced by locally acceptablecommands. Each component is described below, and detailed in thesub-sections that follow.

The context collection system (CCS) 210 of the intrusion detectionsystem 205 captures the interrelated cyber and physical contexts thatenable the detection of legitimate messages and commands that coulddegrade the energy system. The configuration settings, operatingconditions, and other features that make up the context are shaped bycustomer demand, service area, facility size, fuel source, type ofequipment, manufacturer of equipment, age of system, regulations,environmental factors, geographical distribution, and many otherfactors. The context may be assessed both a priori by analyzing itsstatic components as well as in real-time. Context data can betransformed into one unified format which is further refined such thatit can be stored in one or more databases 250 and queried by othercomponents.

The message provenance component 215 and semantic fuzzing component 220use the context information, together with a set of components thatfaithfully model system behavior (i.e., behavior oracles), to determinewhen the monitored energy system starts to exit its normal operationalenvelope. Behavior oracles, or simply oracles, may beindependently-established models of system behavior with varying degreesof cost and fidelity. Oracles may receive actual or hypothetical controlmessages, and parameters regarding the system or its current or paststate (if not already known by the oracle), and provide information on afuture state (such as a prediction of an operational outcome). Asfurther discussed below, behavior oracles may be implemented usinghardware, software, or a combination thereof.

Intrusion detection system 205 may begin by modeling the interactionsbetween devices on different protocols at a system level. This modelingapproach is particularly relevant for attacks in which correlatedsequences of messages can lead to a physical destruction of equipment.The message provenance component 215 builds models of normal behaviorfor the groups of connections established in the network (not just theindependent content of single messages). These models can be used todetect misuse in the history of commands up to the current point, andalso to predict the evolution of the state in response to commandpatterns.

The semantic fuzzing component 220 may then use both the attacker(modeled by a history sequence of control messages up to the currentpoint) and the current state of the system as a concrete seed forprediction of potentially harmful future sequences. In certainconfigurations, a collection of highly-tuned sensors that will bereferred to as behavior oracles can be used to predict possible physicalharm when searching the space of legal message sequences. The output ofthis overall collection and introspection system is threat informationthat relates to both anomalous message sequences and estimated failurestate.

The reporter component 225 may refine the signal received from theprevious layer by merging, de-duplicating, and correlating the threatinformation, capturing it in the STIX/TAXII (threat intelligence)format, and sharing it with an Information Security Operations Center(ISOC) 230. Following standards that are already adopted can enablerapid ingestion and use of threat intelligence by other organizations.

Context Collection

Successfully leveraging context information is challenging due to therichness of the data that can and should be collected. As an example, inthe energy context, an Energy Delivery System (EDS) can be thought of asa network of processes, where each process includes physical components(for example, valves, relays, rotors) and communicating nodes (e.g.,programmable logic controllers (PLCs), Intelligent Electronic Devices(IEDs)) that transfer data to monitor and control these elements. Asevents and phenomena occur in the physical space, they influenceparticular behaviors in the cyber space, and vice versa. For example,when monitoring a process' power flow, the data polling rate dictatesthe traffic's temporal behavior, observable in the cyber context viainter-arrival time between messages. An EDS thus offers rich contextualinformation at both the cyber and physical levels, as well as acontextual cyber-physical relationship governed by the state of theprocess.

Furthermore, the context collection system can collect both static anddynamic data for a more comprehensive view of the actual systemproperties under live operation. This “behavior” loosely defined canoffer the foundation of an independent data source for checking thesemantics of input data and commands against this model of the specificproperties of the system. Static physical context comprises the fixedconstraints of the system, such as the critical ranges and limits ofcomponents and processes that delineate the safe/unsafe operation at thephysical level. The static cyber context includes the defined networkconfigurations like permitted protocols, permitted message types, andfixed network topology. The dynamic context, on the other hand, capturesthe system's status during operation in real-time. This context includesthe condition of the process (e.g., pressure, power level, temperature)and the transient communication patterns (e.g., message rate, observedmessages, transmitting nodes).

Plant systems (e.g., PLCs, Distributed Control Systems (DCSs),Supervisory Control and Data Acquisition System (SCADA), and DataHistorian) can be leveraged as sources to collect physical context. Forexample, integrating with existing platforms can facilitate thecollection of information including process-level actions (e.g.,start-up procedures), physical element status, and alarm history. Forcollecting cyber context, a network flow analysis tool, such as SRI's“SRIFlow” (http://sriflow.csl.sri.com/), may be used to discover enabledprotocol features, flow paths, and other communication information.Because there is likely a large volume of information for context,efforts focus on actionable information that is discrete, timely,predictable, and allows for easy evaluation. Table 1 below provides anexample context for a notional process. To identify the most relevantcontext features, feature extraction and selection methods, such asPrincipal Component Analysis or Fisher Score, may be used to find thefeatures that provide the best accuracy for the models being developed.

Table 1

Table 1 provides an example of context for a rotor control process. Fora process stage, P1, the SCADA system reads the measured pressure andsends setpoints to a PLC controller. The PLC compares the measuredpressure and controls the rotor frequency converter as required to matchthe pressure to the setpoint. The SCADA system (master) sends Modbusrequest messages to the PLC controller (slave) to query the measuredvalues and to change set points, while the PLC controller sends responsemessages containing the measurements or status of the requested change.

Message Provenance

An ICS network may implement multiple protocols, and even a singledevice may interact with other nodes using several different protocols.Messages exchanged between one pair of devices is often correlated toother communicating pairs, and therefore could have an impact on them. Acurrent message can be triggered by a series of events that extend,temporally and spatially, far beyond the previous message in the samesequence. Message provenance aims to understand and model these complexrelationships. In furtherance of this, the intrusion detection systemmay model the device interactions at the system level. This system-levelprovenance model is important for detecting attacks where sequences ofmessages between multiple node pairs can lead to physical destruction ofequipment.

Isolated pairwise sequences of reply and requests in an ICS can beprobabilistically modeled with high accuracy. However, for a system atthe scale of large energy ICSs, e.g., no such conversation takes placeon an island: requests/replies between one pair of devices will beinfluenced by states created through communications between differentpairs. To capture this behavior, the exemplary multi-step processdepicted in FIG. 3 may be used.

In a first step, a message field extractor 305 may be used to safelyisolate the relevant contents (fields) of the messages using, forexample, a LangSec-based (i.e., Language-theoretic Security) parsingapproach. In the ICS context, the most common communicationconfiguration is the master/slave pattern. The master sends commands(with parameters) to the slave, and the slave responds to the commands.The exchange of these command and response pairs depends on both theinternal state of the devices and the different operational modes. Afterunpacking the message contents, a multipair correlation modelingcomponent 310 may attempt to capture the correlations between multiplepairwise connections across the network. The result is a probabilisticmodel 315, based on the techniques detailed below.

Probabilistic Modeling: In contrast to enterprise systems, ICSs(including energy delivery ICSs) tend to exhibit a more constrainedbehavior. ICS systems often have fixed topology, and their specializedfunctionality often exhibits regular communication patterns. Moreover,ICS communication protocols may be simple and not very diverse eachindustry sector has traditionally used a few standard or recognizedcommunication protocols. In the electric power systems, Modbus, IEC(International Electrotechnical Commission) 60870-5-104, IEC 61850, andDNP3 (Distributed Network Protocol) are prevalent (see IT/OT devicecommunications traffic 240 in FIG. 2).

To model message sequences, the intrusion detection system may considerthe sequences of requests and replies generated by regular communicationpatterns. The probability of a command (or data) can be approximated,given each history sequence, facilitating the prediction of the nextsequences of messages. FIGS. 4A and 4B present examples of commandsequences for different devices with regular (FIG. 4A) and less regular(FIG. 4B) patterns. These examples provide patterns captured forindividual devices and connections communicating over Modbus.

The goal is to learn the hidden pattern from a sequence of commands (ordata such as control parameters). Given, for example, the followingsequence of elements: <σ₁, σ₂, σ₁, σ₂, σ₃, σ₃, σ₁, σ₂, σ₁, σ₂, σ₃, σ₃>,the message sequence can be modelled as a first-order Markov chain, thatis:

P(σ^((t))|σ⁽¹⁾, . . . ,σ^((t-1)) =P(σ^((t))|σ^((t-1)))  (1)

The Markov chain model can detect certain abnormal subsequences such as<σ₇, σ₃>, or <σ₂, σ₂>, since those have never appeared in the sequence.However, it is not a good fit for modeling the normal subsequences. Forexample, P(σ₁|σ₂)=P(σ₃|σ₂)=0.5, although <σ₂, σ₁> and <σ₂, σ₃> are stilllegitimate subsequences. However, a Markov chain of order 3 (or more)would learn the above normal subsequences without any ambiguity.

If a sequence of elements is generated by an underlying pattern andexhibits no noise, there exists a minimum order, m, for a Markov chainthat allows for the prediction of the probability of an element bysimply looking at the m most recent elements. The challenge is to buildsuch a model even in the presence of noise, such as legitimatevariations from the base pattern due to missing, out-of-order messages,and/or sporadic tasks. To address this challenge, Probabilistic SuffixTree (PST, or Prediction Suffix Tree), which uses a variable-orderMarkov model representation, may be used. Intuitively speaking, a PSTlearns a set of subsequences of different lengths, e.g. <σ₁>, <σ₂, σ₃>,each of which can be a significant indication of the next element. Thisfacilitates efficient calculation of the probability of the “next”element, without having to look back through all, or a pre-definedlength, of the history. That is,

P(σ^((t))|σ⁽¹⁾, . . . ,σ^((t-1)) ˜P(σ^((t))|σ^((t-1)), . . .,σ^((t-1)))  (2)

for some k that varies depending on σ^((t-1)), σ^((t-2)) and so on.

Consider, for example, the case where four normal Modbus servers (A, B,C, D), are polled by one Modbus client. The polling frequencies for thefour Modbus servers are all different (for example, 0.5, 1, 5, and 10seconds, respectively). The fact that the Modbus client polls thedifferent servers with different frequencies would be captured in thecontext information, enabling determination of the fact that there is acorrelation link between the different connections. If one of theconnections is delayed, the following connections would also be delayed;if sequences in one changes, they perturb the whole high-level sequence.

In the alternative, the intrusion detection system may use neuralnetworks, such as recurrent neural networks (RNN) or other deep learningapproaches, for modeling and predicting the message sequences. RNNs, forexample, are specifically intended for learning sequences of events, butrequire more training data then PSTs.

Role and Integration: The global probabilistic models that are theoutput of the message provenance layer can be used by the intrusiondetection system in at least one of two main ways. First, a model candirectly serve as an anomaly detection mechanism operating on thehistory of commands received so far, and raising an alert if thesequence deviates from the learned norm. This is already a departurefrom current practices, given that the models operate at a system-sidescale. However, to enhance the ability to detect attacks by looking intothe future, and predicting potentially harmful command sequences as theyunfold, a probabilistic message provenance model can serve as an oracle,and predict harmful effects on the system based on the current contextand in response to a sequence of future commands. This will be furtherdiscussed below in the section detailing the operation of behaviororacles.

Semantic Fuzzing

The semantic fuzzing component is an exemplary detection mechanism thatsimulates “speculative execution” of message sequences between ICSdevices. This novel technique generates and tests candidate messagesequences that appear normal and legal, but whose effect is likelymalicious. It may operate independently of actual target devices withoutimpacting their operation. This approach is complementary to the anomalydetection performed by the message provenance subsystem (messageprovenance tests whether currently-observed message sequences areconsistent with the learned model of device interactions, and semanticfuzzing seeks to forecast the future impact of possible follow-onmessage sequences). Semantic fuzzing is, in a sense, an attempt to peerinto the future and anticipate how subsequent messages may impact thesystem and cause it or its subcomponents to drift closer to theiroperational edge (i.e., closer to the upper/lower bounds of the set ofoperational parameters that are deemed acceptably “safe,” or otherwisebehaving in a manner that is unacceptably risky in terms of potentialdamage).

For achieving early detection of such slow state shifts, exemplaryembodiments analyze more than just the past history of commands andinputs, but also extrapolate commands and inputs into the future. Thesystem is able to co-opt the attacker as an oracle to help drive afuzzing process that searches the space of normal message sequences andevaluates the outcome of sending those sequences to the intrusiondetection system's set of High Fidelity Behavior Oracles (320, FIG. 3),thereby taking full advantage of this collection of (a) real, (b)simulated, and (c) theoretically modeled systems.

Semantic fuzzing proactively behaves like an attacker in the sense thatit adopts attacker goals with respect to physical disruption of certainpieces of equipment. A collection of attacker strategies may be encodedas part of the search process. These strategies are domain-specificpatterns for generating unanticipated sequences of messages thatdestructively interfere with normal operation or the physical integrityof the energy system. For example, messages could rapidly oscillatebetween two legal but incompatible settings in order to wear out movingparts or build up friction, pressure, temperature, stress, or strain.Such attack strategies provide templates for achieving certain types ofgoals, but they are not specific enough to encode as a misuse signature,nor is it clear how to assert that any particular message sequence is aninstance of an attacker strategy (modulo an approach like the collectionof behavior oracles). In other words, the presumption of the existenceof valid attacker strategies for breaking or disabling equipment is notan admission that one can somehow a priori guard against thesestrategies merely by writing a signature for a particular messagesequence.

Approach and Architecture. With this insight in mind, a history sequenceof per-device control messages may be used, together with the currentstate of the system (the context), as a concrete seed for prediction offuture sequences that have the potential to cause harm to the system.The semantic fuzzing component (“fuzzer”) can use this seed tosynthesize sequences of valid messages, and then evaluate them against ahierarchy of Behavior Oracles. The oracles (discussed below) estimatethe impact of a sequence on the system, reporting any suspect behaviorto the user, while also reporting this information back to the messagegenerator so that it can refine its search strategy. The overallarchitecture of the semantic fuzzing framework 500 is shown in FIG. 5,with a semantic fuzzer 505 and high fidelity behavior oracles 510.Semantic fuzzing can leverage the unique properties of the domain toconstrain and inform the search.

Message Generator. Fuzzing traditionally refers to feeding (randomlygenerated) test data to the system under test in order to investigateits response. The message generator 515 may use semantic fuzzing as asmarter and more effective way to test a system on a diverse set ofinputs. Exemplary embodiments of the message generator 515 use contextinformation, along with the formal specification of the networkprotocols, as a formal system description that allows the messagegenerator 515 to generate sequences of valid messages 520 (as opposed torandom input data). The formal system description may be expressed, forexample, with a formula ValidMessages in a suitable logical theory, andcan then be efficiently solved by a satisfiability modulo theories (SMT)solver (such as SRI's “Yices”—http://yices.csl.sri.com/), where asolution to the formula corresponds to a sequence of valid messages.

This sequence of messages (the solution) can then be evaluated againstthe intrusion detection system's collection of oracles to estimate howclose the evolution of the system will be to a failing state.Furthermore, the oracles return a set of feedback tokens to the messagegenerator, as a logical formula OracleFeedback, that describes how thisparticular sequence should be changed in order to drive the system evencloser to a failing state. This additional information is then passed onto the SMT solver that can solve for the adapted constraints:

VALIDMESSAGES(m ₁ , . . . ,m _(H) ,x ₁ , . . . ,x _(H))∧ORACLEFEEDBACK(x₁ , . . . x _(L))  (3)

A solution to the above is a new message sequence that takes intoaccount the oracle feedback, and makes the message generationfailure-driven and intrinsically diverse. This technique for test casegeneration and diversification, a sort of concolic testing (aportmanteau of concrete and symbolic), can be very useful in findingmalicious sequences of messages.

Behavioral Oracles

While other modules of exemplary implementations of the intrusiondetection system are responsible for observing the system and generatingpotential malicious behaviors, a set of behavioral oracles evaluates thepotential damage that such behaviors can inflict on the physical partsof the system. The oracles can simulate the physical devicecharacteristics at increasing fidelity levels, and have two mainresponsibilities. First, given a concrete sequence of messages sent to aparticular device, an oracle can predict whether there would be anegative physical impact on the device had this sequence been executed.If the messages are deemed critically harmful, then a definite alarm israised by the intrusion detection system. Otherwise, in order to improvethe coverage that the intrusion detection system can provide, the oraclecan also provide feedback on potential changes that could make the givenmessage sequence even more harmful. The intrusion detection system mayprovide a hierarchical set of oracles, with ranging fidelities (andcorresponding computational complexity). Then, at runtime, depending on,for example, the timing constraints (such as how long it would take fordamage or disruption to occur, a preset time limit available forassessing risk, etc.), the costs and risks of reaching an incorrect riskassessment (e.g., how devastating or disruptive would the maliciousmessages be), etc., a subset of these oracles may be selected so as tomaximize the simulation fidelity. Example oracles that may be includedin various configurations of the intrusion detection system are listedbelow.

Simple Static Context (SSC). In exemplary embodiments, an SSC oracle isa linear collection of facts about the protected equipment. It operatesby comparing the values in the message sequence, and checking if theyare in the allowed range (such as the equipment manufacturer's statedlimits or the configured limits by the asset owner). For example, if therotors of a centrifuge are running at ω_(c)=75,000 rpm, and a commandmessage requires acceleration to ω_(r)=80,000 rpm, which is outside ofthe device limits, then the SSC can report a potential violation.Conversely, if the message requests centrifuge deceleration instead, theoracle could provide as feedback, for example, the formula ω_(r)>80,000. The static context information can also be complemented with anestimated safety envelope computed over the past messages. The safetyenvelope also contains ranges for the values in the message sequences,but with values more conservative than the configured limits, and, inorder to reduce false positives, those can be used instead.

Learned Message Provenance (LMP). An LMP oracle uses the probabilisticmodels described earlier to capture how (possibly not directly related)devices communicate or react to typical events in the monitored system.For example, LMP can isolate the message frequency between two nodes Aand B as an emerging property of the system, and learn that the mean ofthe frequency f is 60 minutes, with a standard deviation of one minute.Given a message sequence m₁, . . . , m₁₀₀ of 100 messages over a span of120 minutes, for example, the LMP oracle may report a change from every60 minutes to every 30 minutes as an anomalous behavior. On the otherhand, if anomalous behavior is not detected, LMP may provide as feedbackthat the sequence might deviate from the expected frequency if anothermessage (say, m₄) between nodes A and B is present. Symbolically, thiscan be expressed by the formula

(m ₄ ·src=A)̂(m ₄ ·dst=B).

Formal Model (FM). An FM oracle maintains mathematical models of thephysical dynamics for devices where this model is either readilyavailable (as part of the device specification), or can be easilydescribed. Formal models can be constructed for discrete controllers ordevices with simple models (e.g., equations relating time, speed,acceleration, etc.). Compared to more computationally expensive oraclesbelow, the formal model is a more efficient mathematical abstractionthat, in addition, also has the means to provide precise symbolicfeedback for improving coverage of possible attacks.

High-Fidelity Simulator (HFS). An HFS oracle is composed of aheterogeneous set of physical equipment simulators. These simulatorsrange from basic ones that already exist for deployment compatibilitytesting, to modified simulators that can model dynamic physicalinteractions (e.g., temperature, pressure, stress, strain). Creating andmanaging an HFS might be computationally expensive, but the benefits arealmost perfect high-fidelity results without the risk of breaking realequipment. In an example, starting from the current process context inthe simulator, and observing the rotor parameters within Process P1according to values (4,000, 80,000, etc.) in the arriving messagesequence, the HFS can directly predict the impact of changing rotorspeeds on the operational speed and process pressure. Oscillating therotor speed would eventually degrade the rotor, cause it to stop, andraise the process pressure beyond the operational envelope. Using theHFS helps the intrusion detection system predict the changes trendingtowards an unsafe condition with each incremental message, even beforethe full sequence of messages is operationalized.

The simulator is capable of modeling emergent interactions/emergentproperties based on physical parameters such as heat, temperature,pressure, strain, stress on all metals and pieces, etc. An existingsimulator may be enhanced, in various configurations, to model a fewcategories of emergent properties, then take the current system state(history) plus current message, and generate (i.e., search) strings oflegal values and parameters in messages (i.e., the search is bounded bythe requirement for legal values and parameters). Messages could bediverted to not only a high-fidelity “honeypot,” but also an automateddiversity farm of honeypots, who would vote if actions would harm them.

Physical Devices (PHY). A PHY oracle is composed of a set of backupdevices that correspond with relatively high-fidelity to the equipmentin the target network. It is a type of honeypot whose physicalproperties approach that of the target network and may thus represent asignificant cost to set up (and pass attacks to, risking breakage). Itmay be the oracle of last resort for tie-breaks, or where previous setsof sensors cannot agree on a decision. While the PHY is ultimately themost precise oracle, its latency and cost position place it at thebottom of the oracle hierarchy exemplary embodiments. It may be decidedthat PHY will not be used except as a last resort to evaluate, forexample, message sequences that all other applicable oracles haveflagged as dangerous.

FIG. 6 provides a conceptual representation of certain exemplaryimplementations of the semantic fuzzing operation 600. A sequence ofactual messages (“S”) observed in the system thus far includes m₀, m₁ .. . m_(i) . . . m_(j) 605 at time_(j) 610. Multiple different messagesequences (“msg seq”) 615 may be generated based on the legal messagesobserved. The message sequences 615 are fed to a cascade of oracles 620that are able to provide an independent cause/effect evaluation of thesequences 615. (It is noted that time is a hidden variable; the sequenceis executed within a time window.) The oracle types depicted in theexample are: (1) Simple Context Comparator (SCC), which makes simplecomparison on context parameters; (2) High Fidelity Simulator (HFS),which simulates the environment and illustrates the impact of themessage sequences on the system's normal operation; (3) Formal Model(FM), which models the context as a set of constraints on differentsystem properties; (4) Learned Message Provenance (LMP), which modelsand predicts the next normal sequences of operations; and (5) Physical(PHY), which uses real physical equipment on which the impact of themessage sequences can be directly observed.

As a simple attack example, a dam includes a sensor that reads waterlevel and a valve that can be opened and closed. An attack may send aseries of Modbus messages with OP_CODE 0x05 with alternating open andclose valve commands. (0XFF00 requests the coil to be ON, 0X0000requests the coil to be off. All other values are illegal and do notaffect the coil.) The valve breaks and overflows the downstream of thedam. The field values for OP_CODE 0x05 are valid; the message sequenceis valid (open/close/open/close/etc.). The semantic fuzzing could easilypredict the following sequence of messages. For example, the PHY, FM,and HFS oracles could detect the physical break in the valve. FM and HFSwill have constraints on physical properties based on context, asdiscussed above.

Reporting

In exemplary embodiments, the threat information received from themessage provenance and semantic fuzzing components may be merged,reduplicated, and correlated to generate threat intelligence that ispackaged, for example, in STIX/TAXII formats. This information may alsobe shared rapidly with, for example, agencies, organizations, and firstresponders, such as the Industrial Control Systems Cyber EmergencyResponse Team (ICS-CERT) and the United States Computer EmergencyReadiness Team (US-CERT) using TAXII, and may be used for correlationwith activities in other utilities and sectors. Moreover, capturing thecontext information that is attached to the threat information, toprovide richer information, would enable operators to develop effectiveremediation plans to avoid significant damage and understand the impacton the operational environment. A tool such as STIXViz (the STIXvisualization tool), which is built specifically for STIX data, can beused to visualize the affected utility's device state, aiding cyberresponders in their remediation activity.

Testing and Fine-Tuning

To test and evaluate exemplary implementations of the intrusiondetection system, the metrics listed below may be used. This list ofmetrics helps characterize the behavior of the protection.

Reduction of risk level. In the energy ICS scenario, the intrusiondetection system seeks to reduce the risk of exposing ICS energydelivery devices to certain types of subtly malicious control messages.Since devices operate in a mode where they receive command and controlmessages on a regular basis, no simple firewall technology or whitelistcan provide protection. Instead, some form of deep introspection shouldbe performed on the traffic in these environments. Reduction of risk forthis class of attacks may be measured using, for example, the securityposture score (e.g., number and type of devices exploited) in thepresence of the intrusion detection system and compare this to the samescenario without the intrusion detection system running. The outputmetric of semantic fuzzing can be particularly helpful to estimate the“distance” from a failure state for given devices and identify a “path”of messages needed to move there, which is directly relevant to systemrisk.

Detection efficacy. Exemplary implementations of the intrusion detectionsystem are an intrusion detection system tuned to a particular class ofattacks in a specific domain. This specificity gives it severaladvantages in demonstrating the ability to detect attacks (to supportthe first metric above, reduction of risk level); this detection abilitycan be measured in several ways: (1) ability for message provenance todetect anomalous communication patterns through measuring false positiveand true positive rates; (2) ability of message provenance to detect theoccurrence of physical events (whether malicious or not), which can bedone as a type of ex post facto check on the sensitivity of messageprovenance to “new” correlated message sequence groups; and (3)completeness and power of discovered message sequences from semanticfuzzing (subsidiary metrics being resources taken by the semanticfuzzing search to identify sequences of particular lengths and impact asmeasured by the behavior oracles).

Nature and amount of context information collected. One significanttemptation for many anomaly detection systems is to attempt to gatherand model all information; this tends to be unfeasible for anysignificantly-sized system and does not necessarily increase detectionaccuracy. The exemplary intrusion detection system instead may focus ongathering context information that has a direct bearing on the (1)patterns of legitimate communication and (2) the physical properties ofequipment most directly related to failure or disruption. Duringtesting, the context collection system may track the nature and amountof context information collected. Experiments may be performed thatassess the impact on detection under varying levels of collected contextinformation (low/medium/high). This can be a useful metric because ithelps asset owners assess how much configuration information they shouldgather in their particular environment to best tune the detection andintrusion detection system.

Exemplary intrusion detection systems discussed above reduce the risk ofexposing devices to a certain type of subtly malicious control messages,and are able to provide low-cost deployment due to the lack ofinterference with the operational environment. This is becausemeaningful data and communications information can be collected in anon-intrusive way. Focusing on the interface between the cyber andphysical layers, exemplary embodiments aim to prevent physical damage toequipment with potential cascading effects.

Equipment need not be patched, upgraded, or modified, since exemplaryimplementations of the system can be deployed alongside existingequipment. The architecture of exemplary intrusion detection systems isnon-disruptive, and can achieve a balance between obtaining access tomeaningful data and communications information. The intrusion detectionmay be deployed as a monitor of communications on an IT/OT subnet (whichmay be well-suited for certain legacy devices), or integrated within newequipment to leverage on-board and strongly compartmentalized computingcapacity. Exemplary implementations do not depend on the nature of anyspecific equipment or protocol because they focus on modelingcommunication patterns, and detection does not depend on observing aspecific feature of current attacks (such as a worm signature). Flexibledeployment options also avoid obsolescence by either being incorporatedinto new equipment or co-residing on network broadcast medium with newor legacy devices. Moreover, the intrusion detection system aims todetect very complex attacks that can evade existing detectiontechniques; these attacks will become more prevalent in the future.

Because the intrusion detection system does not rely on signatures orupdates, it is able to continuously capture the global contextinformation and perform the message provenance detection and semanticfuzzing processes. This helps the system anticipate future trends inboth attacks and equipment. Exemplary implementations have thedeployment flexibility to instrument and monitor an IT/OT network at ornear all known access points, whether wired or wireless; it can examinethe sequence of network messages and complement traditional methods ofauthentication and authorization. The system can supplement theprotection provided by standard strong authentication mechanisms bymodeling and detecting the malicious use of legitimate command sequences(possibly issued by a principal with stolen credentials). That is, thesystem can protect against unauthorized access by detecting carefullycrafted sequences of well-formed, legal messages that mimic anauthorized behavior.

It is noted that operational control systems refer to any systeminvolved in the control of the operation of one or more devices.Examples of such systems include industrial control systems (ICS),related to control systems and associated instrumentation used inindustrial production (such as SCADA systems, DCSs, PLCs, etc.), oftenfound in industrial sectors and critical infrastructures such as energy(e.g., electrical, water, oil, gas, etc.). Operational control systemsinclude control systems used in manufacturing, such as systems thatcontrol the operation of one or more machines, instruments, industrialrobots, etc. in, for example, a manufacturing plant. Further,operational control systems include systems involved in the control ofone or more vehicles or other mobile machines, such as motor vehicles(e.g., cars, trucks, buses, etc.), railed vehicles (e.g., trains, trams,etc.), watercraft (e.g., ships, boats, submarines, etc.), aircraft(e.g., airplanes, drones, etc.), spacecraft (e.g., shuttles, satellites,etc.), or any combination thereof. Such vehicles include, e.g.,autonomous cars and trucks, unmanned ground vehicles (UGVs), remotelyoperated underwater vehicles (ROVs) etc.), and unmanned aerial vehicles(UAV). The control systems may be involved in the operation of devicesthat are for personal, commercial, industrial, military, or other uses.

It is further noted that the “harm” to be avoided need not be limited todamage to a device or disruption to its operation or efficiency. Theharm that may result may be to persons and property, whether in thevicinity of the devices or located remotely. For example, in controlsystems involved in manufacturing, the harm may not be to the actualmachines and industrial robots, but to the products being manufactured(which may be defective, inferior, or otherwise not as intended). In thecase of vehicles, for example, the harm may not be to the vehicleitself, but rather to passengers, cargo, and surroundings. For example,if a safety feature is disabled, rendered less responsive, or otherwisecompromised, the vehicle itself may otherwise remain intact andundamaged (once proper control is restored), but the compromise in thesafety feature (which may be relied upon by the driver who is not awareof the compromise) is harmful because it unnecessarily increases risks,even if a crash does not result.

The present invention has been described in terms of one or morepreferred embodiments, and it should be appreciated that manyequivalents, alternatives, variations, additions, and modifications,aside from those expressly stated, and apart from combining thedifferent features of the foregoing embodiments in varying ways, can bemade and are within the scope of the invention. In the abovedescription, a number of specific details, examples, and scenarios areset forth in order to provide a better understanding of the presentdisclosure. These examples and scenarios are provided for illustration,and are not intended to limit the disclosure in any way. Those ofordinary skill in the art, with the included descriptions, should beable to implement appropriate functionality without undueexperimentation. References in the specification to “an embodiment,” “anexample,” “a version,” “an implementation,” “a configuration,” etc.,indicate that the embodiment, example, version, etc. described mayinclude a particular feature, structure, or characteristic, but everyembodiment, example, version, etc. may not necessarily include theparticular feature, structure, or characteristic. Such phrases are notnecessarily referring to the same embodiment. Further, when a particularfeature, structure, or characteristic is described in connection with anembodiment, it is believed to be within the knowledge of one skilled inthe art to effect such feature, structure, or characteristic inconnection with other embodiments whether or not explicitly indicated.The computerized functionality described above may be implemented inhardware, firmware, software, single integrated devices, multipledevices in wired or wireless communication, or any combination thereof.Computerized functions may be implemented as instructions stored usingone or more machine-readable media, which may be read and executed byone or more processors. A machine-readable medium may include anymechanism for storing or transmitting information in a form readable bya machine. For example, a machine-readable medium may include anysuitable form of volatile or non-volatile memory. In the drawings,specific arrangements or orderings of schematic elements may be shownfor ease of description. However, the specific ordering or arrangementof such elements is not meant to imply that a particular order orsequence of processing, or separation of processes, is required in allembodiments. Further, some connections or relationships between elementsmay be simplified or not shown in the drawings so as not to obscure thedisclosure. This disclosure is to be considered as exemplary and notrestrictive in character, and all changes and modifications that comewithin the spirit of the disclosure are desired to be protected.

Examples

Illustrative examples of the technologies disclosed herein are providedbelow. An embodiment of the technologies may include any one or more,and any combination of, the examples described below.

In an example 1, an intrusion detection method for protecting againstsequences of operationally valid control messages that in combinationharm or disrupt devices in an operational control system includes thesteps of: monitoring operationally valid control messages communicatedin the operational control system to gather current contextualinformation which includes a set of physical constraints on controlsystem properties; determining system-level correlations between controlmessages based on the contextual information; generating sequences ofoperationally valid control messages that would result in actual harmbased on the system-level correlations; and reporting a threat when aharmful sequence of messages is identified.

An example 2 includes the subject matter of example 1, wherein theoperational control system is a manufacturing control system thatcontrols machines used for manufacturing products.

An example 3 includes the subject matter of example 1 and/or example 2,wherein the devices are vehicles.

An example 4 includes the subject matter of example 1, whereingenerating sequences of operationally valid control messages includesusing current messages as starting points and generating subsequentmessages that are predicted to be harmful.

An example 5 includes the subject matter of example 1 and/or 2, furtherincluding evaluating harmfulness of the generated sequences of messagesusing one or more behavior oracles.

An example 6 includes the subject matter of example 1, 2, and/or 3,wherein the one or more behavior oracles evaluate harmfulness bydetermining whether the generated sequences of messages induceoperational drift towards a failure state for one or more devices.

An example 7 includes the subject matter of example 1, 2, 3, and/or 4,wherein the behavior oracles used include a set of hierarchical oracles,and wherein a subset of the hierarchical oracles are used.

An example 8 includes the subject matter of example 1, 2, 3, 4, and/or5, wherein at least two of the hierarchical oracles vary incomputational complexity, and wherein the subset is selected based ontiming constraints.

An example 9 includes the subject matter of example 1, 2, 3, 4, 5,and/or 6, wherein the behavior oracles used include a simple staticcontext (SSC) oracle that is configured to compare a set of values in amessage sequence against an allowable range for the set of values.

An example 10 includes the subject matter of example 1, 2, 3, 4, 5, 6,and/or 7, wherein the behavior oracles used include a high fidelitysimulator (HFS) oracle that is configured to simulate the control systemenvironment to determine effects of message sequences on normaloperations.

An example 11 includes the subject matter of example 1, 2, 3, 4, 5, 6,7, and/or 8, wherein the behavior oracles used include a messageprovenance oracle that is configured to predict subsequent sequences ofnon-harmful control messages.

An example 12 includes the subject matter of example 1, 2, 3, 4, 5, 6,7, 8, and/or 9, wherein the behavior oracles used include a physical(PHY) oracle that is configured to use equipment to directly observe thephysical effects of sequences of operationally valid control messages.

An example 13 includes the subject matter of example 1, 2, 3, 4, 5, 6,7, 8, 9, and/or 10, wherein reporting a threat includes reportinganomalous message sequences or reporting an estimated failure statebased on current messaging.

In an example 14, an intrusion detection system for protecting againstsequences of operationally valid control messages that in combinationharm or disrupt devices in an operational control system, includes aprocessor and a memory having instructions executable by the processor,causing the processor to: monitor operationally valid control messagescommunicated in the operational control system to gather currentcontextual information which includes a set of physical constraints oncontrol system properties; determine system-level correlations betweencontrol messages based on the contextual information; generate sequencesof operationally valid control messages that would result in actual harmbased on the system-level correlations; and report a threat when aharmful sequence of messages is identified.

An example 15 includes the subject matter of example 13, whereingenerating sequences of operationally valid control messages includesusing current messages as starting points and generating subsequentmessages that are predicted to be harmful.

An example 16 includes the subject matter of example 13 and/or 14,wherein the processor is further configured to evaluate the generatedsequences of messages using one or more behavior oracles.

An example 17 includes the subject matter of example 13, 14, and/or 15,wherein the behavior oracles used include a set of hierarchical oraclesthat vary in computational complexity, and wherein the processor isfurther configured to select a subset of the hierarchical oracles basedon timing constraints.

An example 18 includes the subject matter of example 13, 14, 15, and/or16, wherein the behavior oracles used include a high fidelity simulator(HFS) oracle that is configured to simulate the control systemenvironment to determine effects of message sequences on normaloperations.

An example 19 includes the subject matter of example 13, 14, 15, 16,and/or 17, wherein reporting a threat includes reporting anomalousmessage sequences or reporting an estimated failure state based oncurrent messaging.

In an example 20, an intrusion detection method for protecting againstsequences of operationally valid control messages that in combinationharm or disrupt devices in an operational control system includes thesteps of: gathering contextual information by monitoring operationallyvalid control messages communicated in the operational control system,the contextual information including a set of physical constraints oncontrol system properties; mapping message provenance by determiningsystem-level correlations between control messages; using semanticfuzzing to generate predictive sequences of operationally valid controlmessages that would result in actual harm; evaluating harmfulness of thegenerated sequences of messages using one or more behavior oracles; andgenerating a threat report when a harmful sequence of messages isidentified.

1. An intrusion detection method for protecting against sequences ofoperationally valid control messages that in combination harm or disruptdevices in an operational control system, the method including the stepsof: monitoring operationally valid control messages communicated in theoperational control system to gather current contextual informationwhich includes a set of physical constraints on control systemproperties; determining system-level correlations between controlmessages based on the contextual information; generating sequences ofoperationally valid control messages that would result in actual harmbased on the system-level correlations; and reporting a threat when aharmful sequence of messages is identified.
 2. The method of claim 1,wherein the operational control system is a manufacturing control systemthat controls machines used for manufacturing products.
 3. The method ofclaim 1, wherein the devices are vehicles.
 4. The method of claim 1,wherein generating sequences of operationally valid control messagesincludes using current messages as starting points and generatingsubsequent messages that are predicted to be harmful.
 5. The method ofclaim 1, further including evaluating harmfulness of the generatedsequences of messages using one or more behavior oracles.
 6. The methodof claim 5, wherein the one or more behavior oracles evaluateharmfulness by determining whether the generated sequences of messagesinduce operational drift towards a failure state for one or moredevices.
 7. The method of claim 5, wherein the behavior oracles usedinclude a set of hierarchical oracles, and wherein a subset of thehierarchical oracles are used.
 8. The method of claim 7, wherein atleast two of the hierarchical oracles vary in computational complexity,and wherein the subset is selected based on timing constraints.
 9. Themethod of claim 5, wherein the behavior oracles used include a simplestatic context (SSC) oracle that is configured to compare a set ofvalues in a message sequence against an allowable range for the set ofvalues.
 10. The method of claim 5, wherein the behavior oracles usedinclude a high fidelity simulator (HFS) oracle that is configured tosimulate the control system environment to determine effects of messagesequences on normal operations.
 11. The method of claim 5, wherein thebehavior oracles used include a message provenance oracle that isconfigured to predict subsequent sequences of non-harmful controlmessages.
 12. The method of claim 5, wherein the behavior oracles usedinclude a physical (PHY) oracle that is configured to use equipment todirectly observe the physical effects of sequences of operationallyvalid control messages.
 13. The method of claim 1, wherein reporting athreat includes reporting anomalous message sequences or reporting anestimated failure state based on current messaging.
 14. An intrusiondetection system for protecting against sequences of operationally validcontrol messages that in combination harm or disrupt devices in anoperational control system, the intrusion detection system including aprocessor and a memory having instructions executable by the processor,causing the processor to: monitor operationally valid control messagescommunicated in the operational control system to gather currentcontextual information which includes a set of physical constraints oncontrol system properties; determine system-level correlations betweencontrol messages based on the contextual information; generate sequencesof operationally valid control messages that would result in actual harmbased on the system-level correlations; and report a threat when aharmful sequence of messages is identified.
 15. The system of claim 14,wherein generating sequences of operationally valid control messagesincludes using current messages as starting points and generatingsubsequent messages that are predicted to be harmful.
 16. The system ofclaim 14, wherein the processor is further configured to evaluate thegenerated sequences of messages using one or more behavior oracles. 17.The system of claim 16, wherein the behavior oracles used include a setof hierarchical oracles that vary in computational complexity, andwherein the processor is further configured to select a subset of thehierarchical oracles based on timing constraints.
 18. The system ofclaim 16, wherein the behavior oracles used include a high fidelitysimulator (HFS) oracle that is configured to simulate the control systemenvironment to determine effects of message sequences on normaloperations.
 19. The system of claim 14, wherein reporting a threatincludes reporting anomalous message sequences or reporting an estimatedfailure state based on current messaging.
 20. An intrusion detectionmethod for protecting against sequences of operationally valid controlmessages that in combination harm or disrupt devices in an operationalcontrol system, the method including the steps of: gathering contextualinformation by monitoring operationally valid control messagescommunicated in the operational control system, the contextualinformation including a set of physical constraints on control systemproperties; mapping message provenance by determining system-levelcorrelations between control messages; using semantic fuzzing togenerate predictive sequences of legal control messages that wouldresult in actual harm; evaluating harmfulness of the generated sequencesof messages using one or more behavior oracles; and generating a threatreport when a harmful sequence of messages is identified.